{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Part 4c - Loading and Benchmarking\n",
    "\n",
    "In this notebook we will cover the following topics:\n",
    "\n",
    "* Load a trained model from disk\n",
    "* Benchmark the performance of the model on the CPU and GPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.warnings.filterwarnings('ignore')  # Hide np.floating warning\n",
    "\n",
    "import keras\n",
    "\n",
    "from keras.datasets import cifar10\n",
    "\n",
    "# Prevent TensorFlow from grabbing all the GPU memory\n",
    "import tensorflow as tf\n",
    "config = tf.ConfigProto()\n",
    "config.gpu_options.allow_growth=True\n",
    "sess = tf.Session(config=config)\n",
    "\n",
    "import holoviews as hv\n",
    "hv.extension('bokeh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.datasets import cifar10\n",
    "import keras.utils\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "\n",
    "# Save an unmodified copy of y_test for later, flattened to one column\n",
    "y_test_true = y_test[:,0].copy()\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "num_classes = 10\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "# The data only has numeric categories so we also have the string labels below \n",
    "cifar10_labels = np.array(['airplane', 'automobile', 'bird', 'cat', 'deer', \n",
    "                           'dog', 'frog', 'horse', 'ship', 'truck'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the Model\n",
    "\n",
    "Keras provides a `load_model()` function which recreates the model in memory from the provided HDF5 file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import load_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This can take a few seconds depending on how large the model is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "gpu_model = load_model('cifar10_model.hdf5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gpu_model.predict_classes(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Benchmark CPU vs. GPU\n",
    "\n",
    "While the acceleration provided by the GPU is very valuable for training, it is not always necessary when using a trained model for prediction only.\n",
    "\n",
    "Making this tradeoff depends on:\n",
    "\n",
    "* Whether your deployment target has GPUs\n",
    "* What are your throughput and latency requirements?\n",
    "* Can you batch prediction requests, or do you have to process them one at a time?\n",
    "\n",
    "To see how the CPU and GPU performance of our model compares in this notebook, we need to trick TensorFlow into running the model on the CPU even though a GPU is present.  TensorFlow provides a context manager `tf.device()` which can be used to control where operations happen.  If we load the model while the TensorFlow is pinning operations to the CPU device, our model will always run on the CPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with tf.device(\"/device:CPU:0\"):\n",
    "    cpu_model = load_model('cifar10_model.hdf5')\n",
    "    print(cpu_model.predict_classes(x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can compare CPU and GPU performance on 10000 test images (the ideal case for the GPU):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('GPU performance: %d images' % x_test.shape[0])\n",
    "%timeit gpu_model.predict_classes(x_test)\n",
    "print('CPU performance: %d images' % x_test.shape[0])\n",
    "%timeit cpu_model.predict_classes(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on your GPU and CPU, this spread can be 10x or more.\n",
    "\n",
    "Now let's look at the performance for one image:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('GPU performance: 1 image')\n",
    "%timeit gpu_model.predict_classes(x_test[:1])\n",
    "print('CPU performance: 1 image')\n",
    "%timeit cpu_model.predict_classes(x_test[:1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The spread is much smaller, indicating that the CPU might be sufficient for models which have to process inputs one at a time.  (For example, a realtime classifier.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Experiments to Try\n",
    "\n",
    "* Try going back to notebook #3, saving the more complex model to a different file, and loading it here.  How do the CPU and GPU performance compare?\n",
    "\n",
    "If you screw everything up, you can use File / Revert to Checkpoint to go back to the first version of the notebook and restart the Jupyter kernel with Kernel / Restart."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
